Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.

培训最先进模型所需的基础设施变得过于昂贵，这使得培训此类模型仅适用于大型公司和机构。最近的工作提出了几种协作培训此类模型的方法，即通过将许多独立方的硬件汇总在一起，并通过Internet培训共享模型。在此演示中，我们合作培训了类似于Openai Dall-E的文本到图像变压器。我们邀请观众加入正在进行的训练运行，向他们展示有关如何使用可用硬件贡献的说明。我们解释了如何应对与此类训练运行相关的工程挑战（缓慢的沟通，有限的内存，设备之间的性能不均和安全问题），并讨论了观众如何设置协作培训。最后，我们表明所得模型在许多提示上生成了合理质量的图像。

现代深度学习应用程序需要越来越多地计算培训最先进的模型。为了解决这一需求，大型企业和机构使用专用的高性能计算集群，其建筑和维护既昂贵又远远超出大多数组织的预算。结果，一些研究方向成为几个大型工业甚至更少的学术作用者的独家领域。为了减轻这种差异，较小的团体可以汇集他们的计算资源并运行有利于所有参与者的协作实验。这种范式称为网格或志愿者计算，在众多科学领域看到了成功的应用。然而，由于高延迟，不对称带宽以及志愿者计算独特的几个挑战，使用这种用于机器学习的方法是困难的。在这项工作中，我们仔细分析了这些约束，并提出了一种专门用于协作培训的新型算法框架。我们展示了我们在现实条件下的SWAV和Albert预先预价的方法的有效性，并在成本的一小部分中实现了与传统设置相当的性能。最后，我们提供了一份成功的协作语言模型预先追溯的详细报告，有40名参与者。

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered stateof-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. Transformers is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at https://github.com/ huggingface/transformers.

Sky-image-based solar forecasting using deep learning has been recognized as a promising approach in reducing the uncertainty in solar power generation. However, one of the biggest challenges is the lack of massive and diversified sky image samples. In this study, we present a comprehensive survey of open-source ground-based sky image datasets for very short-term solar forecasting (i.e., forecasting horizon less than 30 minutes), as well as related research areas which can potentially help improve solar forecasting methods, including cloud segmentation, cloud classification and cloud motion prediction. We first identify 72 open-source sky image datasets that satisfy the needs of machine/deep learning. Then a database of information about various aspects of the identified datasets is constructed. To evaluate each surveyed datasets, we further develop a multi-criteria ranking system based on 8 dimensions of the datasets which could have important impacts on usage of the data. Finally, we provide insights on the usage of these datasets for different applications. We hope this paper can provide an overview for researchers who are looking for datasets for very short-term solar forecasting and related areas.

Neural networks can be trained to solve regression problems by using gradient-based methods to minimize the square loss. However, practitioners often prefer to reformulate regression as a classification problem, observing that training on the cross entropy loss results in better performance. By focusing on two-layer ReLU networks, which can be fully characterized by measures over their feature space, we explore how the implicit bias induced by gradient-based optimization could partly explain the above phenomenon. We provide theoretical evidence that the regression formulation yields a measure whose support can differ greatly from that for classification, in the case of one-dimensional data. Our proposed optimal supports correspond directly to the features learned by the input layer of the network. The different nature of these supports sheds light on possible optimization difficulties the square loss could encounter during training, and we present empirical results illustrating this phenomenon.

Solar forecasting from ground-based sky images using deep learning models has shown great promise in reducing the uncertainty in solar power generation. One of the biggest challenges for training deep learning models is the availability of labeled datasets. With more and more sky image datasets open sourced in recent years, the development of accurate and reliable solar forecasting methods has seen a huge growth in potential. In this study, we explore three different training strategies for deep-learning-based solar forecasting models by leveraging three heterogeneous datasets collected around the world with drastically different climate patterns. Specifically, we compare the performance of models trained individually based on local datasets (local models) and models trained jointly based on the fusion of multiple datasets from different locations (global models), and we further examine the knowledge transfer from pre-trained solar forecasting models to a new dataset of interest (transfer learning models). The results suggest that the local models work well when deployed locally, but significant errors are observed for the scale of the prediction when applied offsite. The global model can adapt well to individual locations, while the possible increase in training efforts need to be taken into account. Pre-training models on a large and diversified source dataset and transferring to a local target dataset generally achieves superior performance over the other two training strategies. Transfer learning brings the most benefits when there are limited local data. With 80% less training data, it can achieve 1% improvement over the local baseline model trained using the entire dataset. Therefore, we call on the efforts from the solar forecasting community to contribute to a global dataset containing a massive amount of imagery and displaying diversified samples with a range of sky conditions.

计算优化问题解决方案解决方案的雅各布是机器学习中的一个核心问题，其应用程序在超参数优化，元学习，优化为层和数据集蒸馏中的应用程序，仅举几例。展开的分化是一种流行的启发式方法，它使用迭代求解器近似溶液，并通过计算路径区分它。这项工作提供了对梯度下降和Chebyshev方法的二次目标的这种方法的非反应收敛速率分析。我们表明，为了确保雅各布的融合，我们可以1）选择较大的学习率，导致快速渐近地收敛，但接受该算法可能具有任意长的燃烧阶段或2）选择较小的学习率直接但较慢的收敛性。我们将这种现象称为展开的诅咒。最后，我们讨论了相对于这种方法的开放问题，例如为最佳展开策略得出实用的更新规则，并与Sobolev正交多项式领域建立了新的联系。

在从机器人控制到仿真的各种机器人应用中，碰撞检测似乎是规范操作，包括运动计划和估计。尽管该主题的开创性工作可以追溯到80年代，但直到最近，正确区分碰撞检测的问题才成为一个中心问题，尤其要归功于科学界围绕该主题所做的持续和各种努力物理。然而，到目前为止，很少有人提出过解决方案，并且只有对所涉及形状的性质的强烈假设。在这项工作中，我们引入了一种通用和高效的方法，以计算任何一对凸形的碰撞检测的导数，这是通过尤其利用随机平滑技术而显示的，这些技术特别适合于捕获非平滑问题的衍生物。这种方法是在HPP-FCL和Pinocchio生态系统中实现的，并在机器人文献的经典数据集和问题上进行了评估，显示了很少的微秒时间来计算许多真实的机器人应用程序直接利用的信息衍生物，包括许多真实的机器人应用程序，包括可不同的模拟。

强化学习（RL）和轨迹优化（TO）具有强大的互补优势。一方面，RL方法能够直接从数据中学习全球控制策略，但通常需要大型样本量以正确地收敛于可行的策略。另一方面，对方法能够利用从模拟器提取的基于梯度的信息，以快速收敛到局部最佳控制轨迹，该轨迹仅在解决方案附近有效。在过去的十年中，几种方法旨在充分结合两类方法，以获得两全其美的最佳选择。从这一研究开始，我们提出了这些方法的一些改进，以更快地学习全球控制政策，尤其是通过通过Sobolev学习来利用敏感性信息，并增强了Lagrangian技术来实施与政策学习之间的共识。我们通过与文献中的现有方法进行比较，评估了这些改进对机器人技术各种经典任务的好处。